Locality-Sensitive Hashing for Document Similarity Detection

Project Overview

This project focuses on employing Locality-Sensitive Hashing (LSH) to detect document similarities efficiently within a dataset of over 1,500 paragraphs. Through the use of shingling, minhashing, and the banding technique, the project uncovers hidden patterns and pairs of similar documents.

Key Outcomes

• Shingling was implemented to represent text as sets of k-grams.
• Minhashing compactly summarized sets into similarity-preserving signatures.
• Banding efficiently reduced the dimensionality to identify candidate pairs.
• Top five document pairs with the highest similarities were identified.

Tools and Libraries

• Python
• Pandas
• NumPy
• Matplotlib

View the Code

Click the link below to view the full implementation and analysis on GitHub:

View on GitHub